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by Luke E. Kelly, University of Virginia, U.S.A. and Jason Bishop, Beauchamp, 1992). Specifically, motor skill assessment training is 

Northern Michigan University, U.S.A often limited to the teaching of concepts of qualitative assessment 

as opposed to motor skill assessment competency (Walkley & 
Abstract Kelly, 1989). 

This study investigated the effectiveness of a web-based Computer-based instruction (CBI) has been found to be as 
assessment training program and whether explicitly guiding the effective, if not more effective, than traditional teacher-directed 

participants toward the recommended learning sequence would instruction in teaching foundational skills (Huang, Liu, & Chang, 

improve their performance on assessing the underhand roll 2012; Jenks & Springer, 2002; Lee, Shen, & Tsai, 2008). The 

compared to participants that were given free choice of their advantages of CBI over teacher-directed instruction are that it can 

learning sequence. Participants were 48 volunteer pre-service be programmed to adjust the pace and path of instruction based 

undergraduate kinesiology majors from two universities that were on the needs of the learner, it can be accessed 24/7, it can provide 

randomly assigned to two treatment groups. The data collected on immediate feedback, and it can manage and record the learners’ 

the participants included pre and post assessment scores, as well performance. The effectiveness of CBI in education has been 

as program usage data: the frequency and time spent using each of supported by several meta analyses reporting moderate effect sizes 

the training program options. Results revealed a non-significant in favor of CBI. For example, Kulik and Kulik (1991) examined 

treatment effect was found between the two treatment groups, but 254 studies on general use of CBI and found a moderate effect 

significant overall pre to post performance gain of 24.4 percent size (ES = 0.30) while Cohen and Dacanay (1992), examining 

(ES = .772). Program usage data revealed that participants with 47 studies in CBI in health professions, reported an ES of 0.41. 

higher competency scores significantly used the guided practice Additionally, a study by McNeil and Nelson (1991) with 63 studies 

option more frequently and for more time and overall used the involving interactive video and cognitive development found an 
program for more time than participants with lower competency ES of 0.53. 
scores. 

Keywords: physical education, teacher training In response to the need for an innovative way to improve motor 

skill assessment training and the efficacy of CBI, the Motor Skill 
A requisite of teaching motor skills (e.g., kicking a soccer ball, Assessment Program (MSAP) (Kelly, 2007) was developed. The 
shooting a basketball) is the ability to accurately evaluate motor MSAP is a web-based application that provides individualized 
performance and identify what components are correctly performed motor skill assessment training designed based on behavioral and 

and what errors are being made (Horvat, Block, & Kelly, 2007; adult learning theory. Behavioral learning theory was selected 

Kelly, Wessel, Dummer, & Sampson, 2010). Research has indicated because it provides alignment between appropriate learning 

that physical education teachers are not proficient in motor skill strategies (i.e., cues, stimulus, response, feedback, and associations) 

assessment, and often performed no better than undergraduate and the desired goals of MSAP (i.e., foundational knowledge 

students, coaches, or classroom teachers in assessing motor skills of correct motor skill assessment). Behaviorism is based on the 

(Behets, 1996; Biscan & Hoffman, 1976; Hoffman & Sembiante, basic principles of operant conditioning, and learning is associated 

1975; Imwold & Hoffman, 1983; Walkley & Kelly, 1989). with observable changes in behavior. Learning is demonstrated 

Physical education teacher education (PETE) programs have by a correct response after a particular stimulus is presented 

also been shown to be ineffective in teaching PETE students how (Ertmer & Newby, 1993). The primary focus of behaviorism is 

to assess the performance of motor skills (Lounsbery & Coker, “how the association between the stimulus and response is made, 

2008; Pinheiro & Simon, 1992; Reeves, 2000). Typically, PETE strengthened, and maintained” (Ertmer & Newby, 1993, p. 55). In 

programs assume that motor skill assessment skills are learned behavioral learning theory, the presentation of the stimulus and 

when pre-service students complete a theory-based course such as resulting environmental consequence is key, as responses that are 

biomechanics or motor development (Hoffman, 1974; Lounsbery followed by reinforcement are more likely to be repeated in the 

& Coker, 2008). However, research indicates that the theory-based future (Ertmer & Newby, 1993). 

knowledge of movement concepts do not transfer to motor skill The user control and learning sequencing of MSAP were 
assessment skills (Daniels, 1984; Knudson, Morrison, & Reeve, designed using adult learning theory (ALT), which proposes that 

1991). adults have different learning needs than children, and therefore, 

Many barriers have been identified as to why motor skills are programs must be designed to address their particular learning 

not effectively taught in the PETE curriculum, with one of the needs and characteristics (Cercone, 2008). ALT is based on five 

greatest barriers being time (Lounsbery & Coker, 2008). Pre- assumptions that address the difference between children and 

service students are limited by the amount of class time available adults as learners: (a) self-concept of the adult learner is one of 

and the number of courses available, in a typical PETE curriculum, self-directed learning, (b) past experiences of the adult learner 

As such, PETE students do not have adequate opportunities to serve as a resource, (c) readiness to learning is related to changing 

become competent in each topic in their curriculum (Nielsen & social roles, (d) orientation to learning is problem-centered and 
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based on immediate application of knowledge, and (e) motivation 
to learning is by internal factors (Knowles, 1977, 1980; Merriman, 
2001 ). 

MSAP addresses each of the five assumptions of ALT in its 
design. For the first assumption, learners are able to choose and 
control options and the ways of progressing through the MSAP 
modules; resulting in self-directed learning opportunities. For the 
second assumption, learners’ previous experiences of performing, 
teaching, and assessing motor skills are accounted for. Due to these 
past experiences, it is possible that the learner has a background 
and understanding of how to perform motor skills correctly. 
Therefore, learners can chose to view correct performances and 
errors of focal points if needed, but are not required to utilize these 
options if they have an understanding of the focal points from past 
experiences. The third assumption theorizes that adults are ready 
to learn new and required content to address their changing social 
roles. Adult learners will choose to use MSAP because they are at a 
point in their education where they need and understand the benefit 
of becoming proficient in motor skill assessment. The fourth 
assumption is that adults are motivated to learn if they perceive it 
will be applicable and will help them in their real-life situations. 
The MSAP was designed for use by individuals who have a need to 
correctly assess motor skills in their current or future professions, 
including physical educators, pre-service physical educators, 
and kinesiology majors. As motor skill assessment is necessary 
to provide best teaching practices, it is anticipated that these 
learners perceive that the ability to analyze movement patterns is 
directly applicable in their current or future professions. The fifth 
assumption was addressed through the design of MSAP in that adult 
learners would be motivated to use MSAP to improve their motor 
skill assessment because it would be intrinsically rewarding. Adult 
learners should be internally motivated to learn how to accurately 
assess motor skill performance and earn the highest score possible 
on the competency assessments because these are essential skills 
of the profession. 

The MSAP recommended learning sequence (See Figure 1), 
based upon behavioral and ALT, would be that the learner would 
choose to maximize their efficiency by using the tutorial option to 
learn the focal points of the skill and to develop a mental image 
of what each focal point looked like when performed correctly as 
well as some of the common errors made on each focal point. Then 
the learner would use the MSAP guided practice option to develop 
and evaluate their assessment skills. The learner would refer back 
to the tutorial as needed if they consistently had trouble accurately 
assessing one or more of the focal points. Finally, when the learner 
could consistently and accurately assess practice clips in the 
guided practice option in three trials using only real speed, they 
would elect to evaluate their performance using the competency 
assessment option. 

Figure 1. MSAP recommended learning sequence 

Recommended Learning Sequence 

• Learn the Skill 

o Use the Tutorial option to learn the focal points of the 
skill so you can observe them without referring back to 
the prompts. 

o Use the Tutorial option until you have a mental image of 


what the correct focal point performances look like. 

o Review the common errors for each focal point until you 
understand what the common errors look like. 

• Practice 

o Use the Guided Practice option until you can consistently 
and accurately assess students. 

o Start by practicing in slow motion and view the clip as 
many times as needed until you can make an accurate 
judgment. 

o As you practice, identify problem areas and review the 
tutorial on these focal points. 

o As your assessing skill increases, gradually limit your 
practice to real speed and three trials. When you can 
consistently score over 90% using these options, proceed 
to the Competency Assessment option. 

• Competency Assessment 

o The goal of MSAP is to develop your assessment skills so 
that when you take the Competency Assessment you do 
very well - score in the 90's. 

o The Competency Assessment can ONLY be taken 3 times, 
therefore to maximize your performance you should only 
select this option after you have consistently scored well 
using the Guided Practice option. 

o The Competency Assessment is composed of 10 underhand 
roll clips. Each clip can be viewed a maximum of 3 times 
and only in real speed to simulate actual assessment 
conditions in the field. 

o At the end of the Competency Assessment you will be 
given a report that shows your overall accuracy as well as 
your accuracy on each focal point. 

To date, two studies have investigated the effectiveness of 
MSAP. Kelly and Moran (2010) found that MSAP training 
produced a significant pre to post improvements in assessment 
competency and was as effective as teacher-directed training for 
72 pre-service physical education majors. Kelly, Taliaferro, and 
Krause (2012) investigated whether assessment competency 
acquired via MSAP generalized to accurately assessing live 
students in a gymnasium setting. The findings revealed that 36 pre¬ 
service kinesiology majors performed significantly better from pre 
to post (ES = .91) on assessing a motor skill after training using 
MSAP and this training transferred to significant improvements 
(ES = .80) in accurately assessing actual students in a gymnasium 
setting. While these results were very positive, the authors noted 
that informal observations and conversations with some of the 
participants suggested that they may not have been using the most 
efficient learning strategies as suggested by adult learning theory. 
Based on these previous studies, the current study was designed to 
address three research questions. The first question was to confirm 
the effectiveness of MSAP training and to evaluate whether MSAP 
training produced similar findings as reported in previous studies. 
The second question examined whether explicitly guiding the 
participants toward the recommended learning sequence would 
improve their assessment performance of the underhand roll. The 
third question examined how the participants used the MSAP 
options (e.g., tutorial, guided practice, and competency assessment) 
and how this usage corresponded to their overall performance on 
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assessing the underhand roll. This study is significant because it 
evaluates the validity of ALT and whether preservice students act 
according to ALT when they employ the MS AP training options to 
learn how to assess motor skills. 

Method 

Participants 

The participants were a convenience sample of 48 (19 males 
and 29 females) volunteer pre-service undergraduate kinesiology 
majors (age: M = 20.54 years, SD = 1.56) attending two universities 
in the southeast United States. With regard to the number of 
hours of physical education teaching experience, the participants 
reported little prior teaching experience: 91.7% reported 0-20 
hours, 6.3% reported 21-50 hours, and 2.0% reported 51-100 
hours. When asked about prior training in assessing motor skills, 
2.0% reported no prior training, 62.5% reported 1-10 hours, 
29.2% reported 11-25 hours, and 6.3 % reported 26+ hours of 
training. When asked how important they felt it was for physical 
educators to be competent in assessing motor skills 2.0% reported 
they were unsure, 25.0% reported it was important, and 73.0% 
reported it was very important. When asked how comfortable they 
were using technology 2.0 reported being uncomfortable, 25.0% 
reported being unsure, 60.4% reported being comfortable, and 
12.6% reported being very comfortable. Finally, when asked how 
competent they currently felt assessing motor skills, 2.0% reported 
no competency, 27.1% reported being not competent, 39.6% 
reported being unsure of their competency, and 31.3 % reported 
being competent. Participants were informed of the opportunity to 
participate in the study by e-mail. All participants that expressed 
interest were sent a link to access MSAP via the internet. 

Instrumentation 

The research design for this study involved randomly assigning 
the 48 participants recruited from two universities to either the 
guided choice (GC) treatment or the free choice treatment (FC). 
The independent variable was the type of MSAP training received 
by each group. Two versions of MSAP were used in this study. The 
FC group used a version of MSAP based on ALT, which allowed 
them to freely move between the tutorial, guided practice, and 
competency assessment options employing their own learning 
strategy. The GC group used a modified ALT version of MSAP that 
was programmed to guide them towards the recommended learning 
sequence. In this version the recommended learning sequence 
was displayed each time they logged in and the program required 
them to go to the tutorial option once before they could access the 
guided practice option and to use the guided practice option once 
before they could access the competency assessment option. After 
receiving these prompts, they were free to use the options in the 
same manner as the FC group. The primary dependent measures 
for this study were the participants’ pre and post test competency 
MSAP assessment scores. 

MSAP (Kelly, 2007; Kelly & Moran, 2010; Kelly, et al„ 
2012) has three training options: Tutorial, Guided Practice, and 
Competency Assessment. For this study MSAP was designed to 
present training on one motor skill - the underhand roll. In the 
tutorial the participants are taught the focal points of the underhand 
roll and shown video clips of the correct performance as well as 


common errors that participants might see when assessing each 
focal point of the skill. The guided practice option allows the 
participants to practice assessing video clips. Under the guided 
practice option, participants can view a given clip an unlimited 
number of times and in either real speed or slow motion. Then 
the participants assess each focal point by selecting if the focal 
point was performed correctly or not. After they entered their 
assessment, the program provides immediate feedback on the 
accuracy of their assessment (i.e., correct, incorrect) of each focal 
point and the option to view the clip again to see their errors. After 
every five practice clips, the participants are provided a report 
that summarizes their overall assessment accuracy as well at their 
assessment accuracy by focal point. When participants consistently 
demonstrate 90% competency using the guided practice option, 
it is recommended they take the competency assessment. In the 
competency assessment, the participants are shown 10 clips, 
from a dedicated pool of clips, of students performing the motor 
skill. They can only view each of these clips three times and 
only in real speed. After 10 clips are evaluated, participants are 
provided a summative report of their competency. For this study, 
the competency assessment could only be taken a maximum of 
three times and the participants were encouraged to try and get the 
highest score they could. 

The Everyone CAN (Kelly, Wessel, Dummer, & Sampson, 
2010) motor skill assessment item for the underhand roll was used 
for this study. This item consisted of 7 focal points that defined the 
key components of the skill each of which was rated on a binary 
scale (1 or 0) depending upon whether the focal point was correctly 
performed or not. The focal points used for the underhand roll 
were: 

a. Stand with body square to target, weight evenly distributed 
on both feet, feet shoulder- width apart, eyes on target, ball 
held in palm of dominant hand at waist level in front of 
body. 

b. Arm swings back, elbow extended, until dominant hand is 
behind the thigh, with trunk rotation back. 

c. Arm swings forward below the shoulder until dominant 
hand is in front of the thigh, with trunk rotation forward. 

d. Weight shift to the foot on the arm-swing side of the body 
during the arm swing back, and stride forward with weight 
shift to the foot on the opposite side of the body during the 
arm swing forward. 

e. Ball released close to ground, bending hips and knees with 
trunk near vertical, palm facing forward toward target. 

f. Arm follows through well beyond ball release toward the 

target. 

g. Smooth integration (not mechanical or jerky) of the 
previous focal points. 

Procedures 

Participants were sent the URL to access MSAP and were 
informed they could access MSAP 24/7 for one week. When the 
participants logged into MSAP the first time, they read and agreed 
to the IRB consent form, were randomly assigned to either GC 
or the FC groups, and then completed a short survey designed to 
collect information about their experience with assessing motor 
skills and their comfort level using technology. Training ended after 
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the week-long access period or after the participants completed 
their third attempt at the competency assessment. 

Data Collection 

The first time MSAP was used by each participant, the program 
administered a pretest. The pretest was composed of assessing 
10 clips of students performing the underhand roll. Each clip 
could only be viewed three times and only in real speed. No 
formative or summative performance feedback was provided at 
the end of the pretest. During the study the participants could 
take the competency assessment a maximum of three times. The 
competency assessment used the same clips as the pretest from a 
dedicated pool of clips and was administered in the same manner 
as the pretest. At the end of each competency assessment summary 
feedback was provided. This feedback consisted of an overall 
percent accuracy score as well as the percent accuracy achieved for 
each of the seven focal points. The participants’ best competency 
assessment score was used as their post score. The criterion 
assessment ratings for evaluating each clip were determined by a 
panel of five motor assessment experts (Kelly, et ah, 2012) prior to 
the study. Each expert independently rated each of the clips using 
repeated slow-motion and real speed analysis. The results were 
compiled, viewed, and discussed with the experts. After the review 
the experts were asked to rate the clips again which produced an 
overall agreement of 98% for the clips. Given that each test was 
composed of 10 clips and each clip was evaluated based upon 7 
focal points, the maximum score that could be achieved on each 
test was a score of 70. The participants’ scores were calculated by 
dividing the number of focal points assessed correctly, by 70, and 
then multiplying the quotient by 100. 

In addition to recording the test results, MSAP was also 
programmed to record how many times each program option 
(i.e., tutorial, guided practice) was used and for how long. It also 
recorded the total time each participant spent using MSAP. 

Data Analysis 

Three research questions were addressed in this study. The first 
question was to confirm the effectiveness of MSAP training and 
to evaluate whether MSAP training produced similar findings as 
reported in previous studies. The second question examined whether 
explicitly guiding the participants toward the recommended learning 
sequence would improve their performance. The third question 
examined how the participants used the MSAP options, how this 
usage corresponded to their overall performance, and whether 
their usage patterns were consistent with ALT. To address these 
questions descriptive statistics were calculated for all variables 
used in the study. For the first two questions a three factor School 
(university attended) by Treatment (guided or free) by Test (pre 
& post) ANOVA with repeated measures was used. To answer the 
third question the participants’ best MSAP competency score was 
used to stratify all the participants into three performance groups 
(low, middle, and high). The frequency of use and the time spent 
using the MSAP tutorial and guided practice options, as well as total 
time using MSAP were analyzed across these three performance 
groups using a MANOVA. Since total time was a combined 
variable including tutorial and guided practice time, a separate 
ANOVA was run on the total time using MSAP across the three 


groups. In addition, a step-wise regression analysis was conducted 
to determine if any combination of the frequency and temporal 
usage variables could predict post assessment performance. An 
alpha level of .05 was used for all statistical analyses; LSD pair¬ 
wise comparisons were used to interpret group differences, and the 
Eta squared was used to calculated effect sizes. SPSS 19th edition 
was used to perform all the statistical analyses. 

Results 

Descriptive statistics by school and group for the participants’ 
MSAP pre and post competency assessments are shown in Table 
1. Data were graphed and visually inspected to ensure normalicy. 
Homogenity of variance was tested with Levene’s test of equality 
of error variances for both the pre (F(3,44) = 1.507, p = .226) 
and post (F(3,44) = 1.598, p = .203) measures. A three-way 
fixed effects ANOVA with repeated measures was conducted 
comparing the schools and treatment groups on the pre and post 
assessment tests. The only significant main effect found was for 
test (F(l,44) = 149.28, p = 0.000) indicating that the participants’ 
overall performance significantly improved from the pretest to 
the post test. The non-significant main effect for school (F(l,44) 
= .103, p = 0.750) indicated that there was no difference between 
the performances of the participants from the two universities. 
The main effect for treatment (F(l,44) = 3.08, p = 0.086), while 
approaching significance, was not significant indicating that 
the participants that used the guided version of MSAP did not 
perform differently than those that used the free choice version. 
No significant interaction effects were found for treatment by test 
(F(l,44) = 1.05, p = 0.312), school by test (F(l,44) = 3.40, p = 
0.072) or school by treatment by test (F(l,44) = 2.33, p = 0.134). 


Table 1. MSAP Pre and Post Competency Scores by School 
and Treatment Group (Guided Choice and Free Choice) 


Test 

School 

Group 

N 

Mean 

Std. Deviation 

Pre 

1 

GC 

15 

62.87 

13.39 



FC 

14 

50.36 

6.42 



Total 

29 

56.83 

12.22 


2 

GC 

10 

51.90 

10.52 



FC 

9 

51.89 

6.99 



Total 

19 

51.89 

8.78 


Total 

GT 

25 

58.48 

13.27 



FC 

23 

50.96 

6.53 



Total 

48 

54.88 

11.15 

Post 

1 

GC 

15 

79.73 

10.71 



FC 

14 

78.21 

13.28 



Total 

29 

79.00 

11.83 


2 

GC 

10 

83.30 

7.97 



FC 

9 

81.11 

12.08 



Total 

19 

82.26 

9.90 


Total 

GC 

25 

81.16 

9.69 



FC 

23 

79.35 

12.63 



Total 

48 

80.29* 

11.11 

*The overall pre to Dost change was significant d <.05 
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Finding no significant school or treatment main effects the 
total sample was combined and then stratified by their post 
competency assessment scores to form three performance groups: 
low (cut-off score < 77) performance group (LPG), middle (cut¬ 
off score < 87) performance group (MPG), and high (cut-off score 
>88) performance group (HPG). The performance groups were 
compared on their MSAP usage in terms of their frequency of use 
of the tutorial and guided practice options and time spent using the 
tutorial and guided practice options as well as the total time spent 
using MSAP. Descriptive statistics for the performance groups 
MSAP usage are shown in Table 2. The MANOVA analysis of the 
MSAP frequency and temporal variables across the performance 
groups was significant F(12,80) = 9.82, p < .000; Wilk’s A = .164, 
partial T| 2 = .60. Significant univariate differences were found 
between the performance groups on frequency of using the guided 
practice option (F(2,45) = 4.92, p = 0.012, ES = .18) and time 
using the guided practice option (F(2,45) = 8.32, p = 0.001, ES = 
.27). LSD pair-wise comparisons were conducted to interpret the 
performance group differences on these variables. For frequency 
of guided practice usage it was found that FIPG used the guided 
practice options significantly more than the LPGbut not significantly 
more than the MPG. For the time in minutes using guided practice, 


Table 2. Descriptive Statistics for MSAP Usage by 
Performance Group 


Variable 

Group 

N 

Mean 

Std. Deviation 

Post Competency 

Low 

16 

66.8 

7.77 

(Score) 

Medium 

16 

83.8 

3.41 


High 

16 

90.3 

1.44 



Total ■ 

48 80.3 

11.11 

Tutorial Frequency 

Low 

16 

1.63 

0.81 

(# times used) 

Medium 

16 

2.06 

1.53 


High 

16 

2.37 

1.09 


Total 

48 

2.02 

1.19 

Tutorial Time 

Low 

16 

5.06 

4.79 

(Minutes) 

Medium 

16 

8.13 

7.39 


High 

16 

7.31 

4.87 


Total 

48 

6.83 

5.84 

Guided Practice 

Low 

16 

3.56 

4.10 

Frequency 

Medium 

16 

8.00 

6.85 

(# times used) 

High 

16 

15.06* 

16.27 


Total 

48 

8.88 

11.30 

Guided Practice 

Low 

16 

13.94 

13.36 

Time 

Medium 

16 

36.38** 

25.29 

(Minutes) 

High 

16 

64.00*** 

52.99 


Total 

48 

38.10 

39.82 

MSAP Total Time 

Low 

16 

19.00 

14.79 

(Minutes) 

Medium 

16 

44.50** 

27.54 


High 

16 

71 31*** 

53.45 


Total 

48 

44.94 

41.10 

*HPG was significantly (p < .05) better that the LPG 

**MPG was significantly (p < .05) better that the LPG 

**HPG was significantly (p < .05) better that the MPG & LPG 


the FIPG significantly used this option for more time than both the 
MPG and the LPG, and the MPG used it significantly more than 
the LPG. The ANOVA results on MSAP total time usage was also 
significant (F(2,45) = 8.57, p = 0.001, ES = .28). For total time in 
minutes using MSAP, the FIPG significantly used this MSAP for 
more time than both the MPG and the LPG and the MPG used 
it significantly more than the LPG. Finally, a stepwise regression 
analysis was conducted to determine if any combination of these 
variables would predict post competency performance. Total time 
using MSAP significantly predicted MSAP performance, b = 
.526, t = 4.19, p < .000. Total time using MSAP also explained a 
significant amount of variance in MSAP performance, R2 = .277, 
F(l,46) = 17.60, p <.000, y= 73.901 + (.142 X Total Time). 

Discussion 

The overall competency level of the participants on the 
pretest (M=56.8) is consistent with the literature indicating that 
preservive kinesiology majors were not proficient in assessment 
(Behets, 1996; Biscan & Floffman, 1976; Floffman & Sembiante, 
1975; Imwold & Hoffman, 1983; Walkley & Kelly, 1989). The 
significant main effect for test in the present study supports 
previous studies using MSAP that reported significant pre to post 
assessment improvements for pre-service kinesiology majors on 
learning to assess motor skills (Kelly & Moran, 2010; Kelly, et 
al., 2012). Kelly et al (2012) found a 28.3 percent improvement 
(ES = .91), which is comparable with the 25.41 percent improvement 
(ES = .77) found in this study. 

The current study compared two versions of MSAP. One 
version, used by the GC group, was designed to provide the 
participants more guidance on how to effectively use the MSAP 
options with the goal of increasing the efficiency of their learning. 
The FC group used the standard version of MSAP that gave the 
participants complete freedom to use any of the program options. 
This change was employed based on anecdotal comments from 
participants in previous studies that suggested that at least some 
of these participants spent most of their time learning by trial and 
error in guided practice option and did not take advantage of the 
guided practice feedback by using the tutorial options. The group 
main effect was found to be non-significance (F(l,44) = 3.08, p = 
0.086), but was approaching significance. Cautiously examining 
this difference (See Figure 2) reveals that although the participants 
were randomly assigned to group, the GC began higher on the 
pretest (GC: M = 58.48; FC: M = 50.96) and finished only slightly 
higher on the post assessment (GC: M = 81.16; FC: M = 79.35) 
with no significant differences between groups on the pre and 
post measures. Overall, the GC showed a 22.68% improvement 
compared to a 28.39% improvement by the FC, indicating that any 
group effect favored the FC. These findings indicate that explicitly 
informing the participants in the GC of the recommended learning 
sequence and requiring them to use both the tutorial and guided 
practice options at least once before taking the competency 
assessment did not appear to enhance their performance compared 
to the FC and may have potentially hindered it. Further research 
is warranted to investigate whether not treating participants as 
adult learners and constraining their access to the program options 
actually hinders their performance. 
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Treatment by Pre/Post Comparison 

100 I- 



Pre Score I Post Score 


■ Guided Choice 

58.48 

81.16 

■ Free Choice 

50.96 

_79.35 


Figure 2. MSAP pre/post comparison by treatment group 


It appears that forcing the participants in the GC to at least 
review the recommended learning sequence and use the tutorial and 
guided practice options once did not enhance their performance. 
Although the participants in the FC appear to have performed 
better, this does not necessarily mean they were acting like adult 
learners. The third question investigated in this study examined 
how the participants used the MSAP options and how these usage 
patterns corresponded with their overall performance. Not finding 
significant school or treatment group effects, the total sample was 
stratified by their post competency assessment score and divided 
into three performance groups of 16: low, middle, and high. The 
groups were then compared on their frequency and temporal usage 
of the tutorial and guided practice options as well as their total 
time using MSAP. The findings revealed that overall there was 
no difference between the three groups in their use of the tutorial 
option, which was used on average only two times and for a total 
of 6.8 minutes by each participant. Significant differences were 
found by group for both frequency and time usage of the guided 
practice option. The HPG used the practice option significantly 
more frequently than the LPG and approached significance with 
the MPG (p = .063). For time, the HPG significantly used the 
guided practice option for more time than both the MPG and the 
LPG and the difference between the MGP and LPG approached 
significance (p = 0.075). Finally, for overall time using MSAP the 
HPG used MSAP for more time than both the MGP and the LPG 
and the MPG used MSAP for significantly more time than the 
LPG. On a positive note, these findings indicate that participants 
in the HPG were able to achieve 90+% competency after using 
primarily the guided practice option for a little more than an hour 
(M = 64 minutes) and for the MGP to achieve 84% accuracy after 
only 45 minutes. These results demonstrate that the guided practice 
option in a trial and error fashion can produce significant gains in 
competency in relatively short periods of time. 

The dominant use of the MSAP guided practice option is 
consistent with previous informal observations and anecdotal 
comments from past participants. The concern is that the 
participants are not taking advantage of the tutorial information 
that explicitly defines each focal point and provides video examples 
of the conect performance. While the current results support that 
one can clearly learn by just using the guided practice option, it is 
not the most effective means since it only focuses on identifying 


the participant’s errors and does not show them what the correct 
performance should look like. After participants evaluate a clip 
in guided practice they receive immediate feedback on which 
focal points they judged as correct and which ones were judged 
incorrectly. They then have the option to view the clip again to see 
their errors. 

After every five clips, they are given a summative report that 
indicates their overall percent accuracy as well as their percent 
accuracy for each focal point. For any focal point where their 
percent accuracy is less than 60% they are encouraged to return to 
the tutorial and review the correct performances and explanations 
with the goal of developing a better mental image of what that 
correct performance of the focal points should look like. Given 
that the participants on average only used the tutorial two times 
for a total of 6.8 minutes suggests that the participants were not 
fully taking advantage of this option to facilitate the efficiency of 
their learning. This is particularly noteworthy and hard to explain 
in light of the fact that the GC group was explicitly informed of the 
recommended learning sequence and had to go to the tutorial once 
when they initially used MSAP. 

There are several other implications and recommendations for 
future research that can be drawn from the MSAP usage data. While 
the participants were acting in some ways like adult learners in that 
they were choosing their own learning path, from a design and 
learning efficiency perspective they were not taking full advantage 
of the program options or using the most efficient learning path. 
More research is needed to investigate whether this usage pattern is 
unique to pre-service college-aged participants and whether other 
groups, such as in-service physical education teachers, would use 
MSAP differently. 

It is recommended that future studies using MSAP with both pre¬ 
service and in-service participants conduct follow up interviews 
and/or focus groups after the training phase to investigate what 
strategies the participants were actually using and to explore 
specifically why and how they chose to use the tutorial option. 
The five assumptions of ALT (Merriman, 2001; Knowles, 1977, 
1980) should be used as the basis for formulating questions to 
determine the degree to which these different participant groups 
meet these assumptions. These findings should then be used to 
revise and refine MSAP so that it can achieve the greatest degree 
of competency in the shortest amount of time. This is an important 
consideration given the large number of physical and motor skills 
a physical educator must be able to competently assess. 

As technology advances continue and the cost effectiveness 
of distance education increases, increasingly more PETE will 
become computer based. It is therefore important to understand 
how learners interact with different forms of computer delivered 
training so that these applications can be designed to optimally 
address the needs of these learners. 
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